Skip to content

feat: ESI rate-limit overhaul Phase 2+3 — eveapi#670

Open
herpaderpaldent wants to merge 6 commits into
4.xfrom
feat/esi-rate-limit-overhaul
Open

feat: ESI rate-limit overhaul Phase 2+3 — eveapi#670
herpaderpaldent wants to merge 6 commits into
4.xfrom
feat/esi-rate-limit-overhaul

Conversation

@herpaderpaldent
Copy link
Copy Markdown
Contributor

@herpaderpaldent herpaderpaldent commented May 10, 2026 •
edited
Loading

Summary

Phase 2+3 of the ESI rate-limit overhaul (depends on esi-client PR #22).

Critical bug fix (Phase 2)

429/420 responses were permanently killing jobs — handleException() called fail() on all 4xx, but 429 and 420 should be retried. This is live now with ESI's floating-window rate limit system.

Phase 2 Changes

Bug fixes

  • RetrieveFromEsiBase: catch EsiRateLimitedException/EsiErrorLimitedException before RequestFailedException → call release($retryAfter) instead of fail()
  • handleException(): ServerException (5xx) now respects Retry-After header for delay

New: EsiJobReleasedException sentinel

  • Thrown after release() to stop executeJob() body from continuing
  • Caught silently in EsiBase::handle() — no noise in error reporting

New: EsiProactiveRateLimitMiddleware

  • Reads Redis bucket state before job runs
  • If remaining < 10% of limit, calculates delay from refill rate and releases job proactively
  • recordResponse() stores rate-limit state to Redis after each successful ESI call
  • Prevents the costly 5-token penalty of actually hitting a 429

EsiBase improvements

  • $tries: 3 → 10 (rate limits need many retry cycles)
  • backoff(): extended to 9 entries (15min cap)
  • Added EsiProactiveRateLimitMiddleware to middleware()

EsiRequestContainer + RetrieveEsiData

  • Added ?string $compatibility_date = null propagation through to EsiConfiguration

collect($response) → collect($response->data)

Updated 15 files after EsiResponse dropped ArrayObject inheritance (Phase 1 change):

  • All asset, wallet, skills, mail, contacts, contract, corporation jobs

Phase 3 Changes — JobChecker future-proofing

checkVersion()

  • Gracefully handles missing x-alternate-versions (no more undefined index crash)
  • Warns when endpoint uses x-compatibility-date (new ESI versioning scheme CCP is planning)
  • Warns when endpoint has neither field

checkMiddleware()

  • Now requires EsiProactiveRateLimitMiddleware in all jobs

checkRateLimit() (new check)

  • Verifies EsiProactiveRateLimitMiddleware is present when endpoint declares x-rate-limit extension (future ESI spec field)

ResolveUniverseStructureByIdJob

  • Fixed: was missing EsiProactiveRateLimitMiddleware (overrides middleware() independently)

Test results

  • 524 passed, 1 failed (pre-existing MigrateDbTest — MySQL not available in dev env)
  • PHPStan: ✅ No errors
  • Type coverage: ✅ 100%
  • seatplus:check:endpoints: ✅ All 36 jobs pass

Dependencies

⚠️ Requires esi-client PR #22 to be merged first. A path-repository entry in composer.json points to the local esi-client for development — remove it after PR #22 is published.

Remaining after merge

  • Corporation Projects ESI endpoint jobs (blocked — endpoints not in live ESI spec yet)
  • SDE REST API client integration (blocked — needs correct REST API base URL)

herpaderpaldent and others added 3 commits May 10, 2026 20:53
@herpaderpaldent
feat: ESI rate-limit overhaul Phase 2 — eveapi
4131e18 
- Fix handleException(): 429→release, 420→release, 5xx→release with Retry-After
- Add EsiJobReleasedException sentinel (silent catch in handle())
- EsiBase: tries 3→10, backoff extended, EsiProactiveRateLimitMiddleware added
- EsiRequestContainer: add compatibility_date propagation
- RetrieveEsiData: wire compatibility_date to EsiConfiguration singleton
- collect($response) → collect($response->data) in 15 job files (ArrayObject removed)
- EsiProactiveRateLimitMiddleware: proactive token-bucket throttling via Redis
- composer.json: path repository for local esi-client dev (remove after PR #22 merges)
- .gitignore: exclude .phpunit.cache/

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@herpaderpaldent
feat: Phase 3 — JobChecker future-proofing and EsiProactiveRateLimitM…
8896cc7 
…iddleware enforcement

- JobChecker.checkVersion(): gracefully handle missing x-alternate-versions;
  warn when endpoint uses x-compatibility-date (new ESI versioning scheme)
- JobChecker.checkMiddleware(): require EsiProactiveRateLimitMiddleware in all jobs
- JobChecker.checkRateLimit(): new check for x-rate-limit spec extension (future ESI)
- JobChecker.checkJob(): add checkRateLimit() to pipeline (7 checks total)
- ResolveUniverseStructureByIdJob: add EsiProactiveRateLimitMiddleware to middleware()
- JobCheckerTest: 5 new tests (compatibility-date×2, rate-limit×3), 1 new middleware test

All 36 jobs pass seatplus:check:endpoints ✓

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@herpaderpaldent
fix: correct .gitignore (separate composer.lock/.phpunit.cache lines)
d42a319 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@herpaderpaldent herpaderpaldent changed the title feat: ESI rate-limit overhaul Phase 2 — eveapi feat: ESI rate-limit overhaul Phase 2+3 — eveapi May 11, 2026
herpaderpaldent and others added 3 commits May 11, 2026 08:23
@herpaderpaldent
chore: bump PHP to ^8.5, upgrade to pest v4, fix larastan 3 issues
f23011d 
- Require PHP ^8.5 (targeting current stable)
- Upgrade pest, pest-plugin-laravel, pest-plugin-type-coverage to ^4.0
- Upgrade larastan to ^3.0 (requires configDirectories in phpstan config)
- Upgrade phpstan-deprecation-rules and phpstan-phpunit to ^2.0
- Upgrade rector to ^2.0, driftingly/rector-laravel to ^2.0
- Add configDirectories: [config] to phpstan.neon.dist (larastan 3 requirement)
- Generate phpstan-baseline.neon for pre-existing larastan 3 strictness errors
- Remove dead catches for replaced exception types in RetrieveEsiData
- Add typed class constants (const string/int/float/array) throughout
- Add FORK_MEM_PER_PROC workaround for pest-plugin-type-coverage v4 + PHP 8.5
  parallel fork race condition (known upstream bug)
- Update esi-client constraint ^3.0 → ^4.0 and re-add path repo for local dev
- Update CI workflows: php-version 8.3 → 8.5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@herpaderpaldent
feat!: migrate all jobs from EsiResponse magic access to ->data->prop…
eed34f9 
…erty

Removes reliance on the now-deleted __get/__isset shim in esi-client v4.
All job executeJob() methods now explicitly access the decoded response
body via $response->data->property instead of the deprecated magic proxy.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@herpaderpaldent
feat: Phase 0.5 - typed Response DTOs for all ESI jobs
08ba91a 
- Add 35 readonly class DTOs in DataTransferObjects/Responses/
  organized by subdomain (Alliances, Assets, Character, Contacts,
  Contracts, Corporation, Killmails, Mail, Skills, Universe, Wallet)
- Each DTO has from(object $data): self factory; optional ESI fields
  use nullable defaults
- Update all jobs to use typed DTOs instead of stdClass magic access:
  AllianceInfoJob, CharacterAssetJob, CharacterAssetsNameJob,
  CharacterAffiliationJob, CharacterInfoJob, CharacterRoleJob,
  CorporationHistoryJob, CharacterContractsJob, ContractItemsJob,
  CorporationDivisionsJob, CorporationInfoJob, CorporationMemberTrackingJob,
  KillmailJob, MailBodyJob, MailHeaderJob, SkillQueueJob, SkillsJob,
  all 8 Universe jobs, CharacterBalanceJob, CorporationBalanceJob,
  WalletJournalBase, WalletTransactionBase
- Update ProcessContactResponse and ProcessContactLabelsResponse services
- Remove all optional(), data_get(), property_exists() calls on ESI data

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@herpaderpaldent